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This paper discusses the building of a hypermedia-based 
reference station at the Wright Laboratory Technical Library, 
Wright-Patterson Air Force Base, OH. A rapid prototyping 
approach is used in the system's development in that it is based 
upon continuous library user input. Included is a brief 
discussion of methodology and procedures, a literature review, 
and project background. Following this, the paper focuses on an 
electronic user suzrvey from which data is collected and an 
analysis is made. Challenges and obstacles to the administration 
of the survey instrument are also discussed. A conclusion is 
then drawn based upon the efforts of the previous year. The goal 
of this project is to facilitate access to information both 
within and beyond th^^ physical bounds of the library, thus 
resulting in a higher level of service to users. 
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INTRODUCTION 

Computerized user systems are commonplace in libraries 
today. Online public access catalogs, CD-ROM workstations, and 
library orientation systems are but a few of the applications 
that have come about as a result of technological advances. A 
more recent application that has appeared in libraries is the 
hypermedia-based orientation/reference station. How well these 
stations meet user needs is of crucial interest to the libraries 
that design and maintain them. Obtaining user input on a 
constant basis and making modifications accordingly is an 
approach that can be used to achieve the best "fit" in terms of 
meeting needs. 

OBJECTIVE 

The objective of this research and development project is to 
incorporate user input into the design and maintenance of a 
hypermedia-based reference station which will facilitate access 
to resources both within and beyond the physical boundaries of 
the library. 
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PROCEDURES 



A literature search in several areas was conducted from July 
1993 to June 1994* These areas included hypertext and hypermedia 
history, human-machine interaction theory as it relates to 
hypermedia, hypermedia stations already placed in libraries, 
rapid prototyping, and the evaluation of the effectiveness of 
both human and automated reference service. The prototype 
reference station, named "AskMe" which has also been under 
development since July 1993, was introduced into the Wright 
Laboratory Technical Library at Wright-Patterson Air Force Base, 
OH in mid-April 1994. A study was conducted in which an online 
survey instrument was made available on AskMe from May 16 to July 
11. Data was collected and tabulated during this time. This 
paper details these project activities as well as provides a 
review of background literature and a discussion of future plans. 



I 



3 

LIMITATIONS 

The study discussed in this paper comprises but one phase of 
the rapid prototyping process. This phase involves the 
collection electronically of user data from one site, that being 
the Wright Laboratory Technical library • 

DEFINITION OF TERMS 

"Reference station" refers to a microcomputer-based system 
designed to aid library users in accessing information contained 
both within and outside of a library • 

"Hypermedia" is defined as text, graphics, video, and audio 
linked in such a way as to allow a user to access information 
easily and non-sequentially . 

"Rapid prototyping" is the design and presentation of many 
trial versions of a product in a short period of time based upon 
continuous interaction between developer and customer* 
"Effectiveness" is defined as the level of user 
satisfaction with the system's performance in either answering a 
question or leading one to an answer, providing such an answer 
indeed exists* 
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METHODOLOGY 

AskMe has been set up at the Wright Laboratory Technical 
Library at Wright Patterson Air Force Base^ OH* The computer 
hardware consists of a Unisys 386-based PC with a 14" VGA color 
monitor. A keyboard and mouse are also provided* A larger 
monitor was desirable but none was available. At the first 
opportunity^ a 16-19" monitor will be substituted. The software 
platform consists of ToolBook v 1.5 running on Windows 3.I. 
ToolBook was chosen as a result of its favorable mention in the 
literature and its reasonable price. Presently^ the AskMe 
application is mounted on a DECnet common drive using Runtime 
ToolBook and the required network program that is also provided 
with ToolBook. As a result^ AskMe is available to users in the 
library while enhancements to the system can be developed at and 
implemented from another site in-house. These enhancements will 
be based upon feedback from users concerning how effectively this 
reference station is able to ansrwer questions outright or lead to 
other sources that will provide answers. 

Anonymous input from users concerning AskMe 's effectiveness 
has been gathered by means of an eleven-question electronic 
survey instrument. A similar approach to administering an 
instrument was used in a study concerning the U-Search* CD-ROM 
service at the University of North Carolina Davis Library. 1 This 

^Tim Bucknall and Rikki Mangrum^ "U-Search: A User Study of 
the CD-ROM Service at the University of North Carolina at Chapel 
Hill," RQ 31 (Summer 1992): 542-554. 
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method was chosen for the study because it required no extra 
human resources and could be utilized In silence . The Wright 
Laboratory Technical Library has a small staff which serves a 
relatively large user community, thus methods involving formal 
observation and recording of data would have been highly 
disruptive in terms of service to this community* In addition, 
any methods such as formal interviews, besides consuming time and 
already over-extended staff resources, would have been a 
potential disturbance to customers. 

The AskMe survey is accessible by clicking on a button in 
the upper right-hand corner of any screen. Data has been 
collected electronically and then placed into text files named 
according to the date of retrieval. Modifications designed to 
increase the effectiveness of the reference station will be made 
based upon analysis of this input. 
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LITERATURE REVIEW 

Hypermedia 

Within the past few years, reference stations have sprung up 
in libraries nationwide. This option has been made more 
attractive by the development of hypermedia. The texia hypermedia 
is an expansion of the term hypertext. Both terms were coined by 
Ted Nelson in 1965. Hypertext has been defined by him as "non- 
sequential writing — text that branches and allows choices to the 
reader, best read at an interactive screen. "2 while hypertext 
refers to textual infoxrmation, hypermedia includes graphics, 
video, and audio as well as text. 

The concept of non-sequential organization of materials can 
be traced back to 1945 when Vannevar Bush described the "memex" 
which would have allowed one to retrieve interrelated miniatur- 
ized documents stored inside a mechanical/electrical desk.*^ The 
idea of the memex stayed with Bush throughout the rest of his 
life, and in his book Pieces of the Action he stated that "The 
day is not yet here, but has come far closer."^ 

At this time. Nelson was already at work on his Xanadu 

project. This undertaking involved the development of a system 

^Theodore Holm Nelson, Literary Machines 90.1 (Sausalito CA: 
Mindful Press. 1990), 0/2. 

^Vannevar Bush, "As We May Think," Atlantic Monthly, 
176 no. 1, (July 1945): 101-108. 

^Vannevar Bush, Pieces of the Action (New York: Morrow, 
1970), 190. 
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. which would allow one not only to access and produce electronic 
documents but also to link them together. Xanadu is still under 
development today, combining hypertext and networking principles • 

Another important figure in the history of hypermedia and, 
for that: matter, the history of human-computer interaction, was 
Doug Engelfaart, the inventor of the mouse. ^ From the late 50s, 
his work at Stanford Research Institute has increased our 
understanding of how computers can be used as an extension of the 
human in performing tasks. Engelbart's device is presently the 
central tool in most hypermedia systems. 

As technological advances improved the power of computer 
systems, the fascination with hypertext increased. By the mid- 
1980s, Apple had introduced HyperCard for its Macintosh line of 
microcomputers. This brought hypermedia capabilities into the 
schools and libraries. The first use of HyperCard in a library 
setting is believed to be by Marty Kesselmen at Rntgers 
University. In 1984, he used a pre-released version of HyperCard 
to develop LSM InfoMaster, a library instruction program still in 
use today. ^ 

By the late 80s, hypermedia was being utilized throughout 
the library environment for instructional and reference uses. 



^Robert A. Szymanski and others. I ntroduction to Computers 
and Information Systems (Columbus, Toronto, Lomdom, Melbourne: 
Merrill Publishing Company, 1988), 71. 

^Susan Borden am. Margaret Osburn, "The Video File: 
Interactive Marketing for the Library Part 2," Marketing 
Treasures 6 no. 5, (May/June 1993): 3. 
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Presently, systems are in place, being set up, or planned in many 
libraries. Internet listserves such as LIBREF-L have contained 
requests for information and advice from those seeking to set up 
such hypermedia systems. A company, TTSS (Rockville, MD), has 
developed the Self Inform electronic directory system which 
utilizes text, graphics, and video to provide information to 
users. ^ Libraries that use Self Inform include the University of 
Tennessee, the University of California at San Diego, and Los 
Angeles Public Library. 

Through this period including the present, many hypertext 
and hypermedia authoring systems have made their appearance. 
Commercial Products such as ToolBook, HyperWriter, Hyperties, 
Linkway, and shareware such as HYPER and HyPER2 have been and/or 
are being used. Other software packages that have hypertext 
capabilities include AskSam, Knowledge Pro, and Owl Guide. ^ The 
University of Texas-Pan American Library has set up a hypermedia- 
basf^d system using Guide. ^ 

ToolBook, released in early 1990,, was the first product of 
Asymetrix Corp. The company, located in Redmond, WA, was founded 
by Microsoft co-founder Paul Allen in 1985. Billed as a 
"software construction set", ToolBook was to be released at the 

^TTSS, TTSS News of '93 (Rockville MD: TTSS, 1993): 

1-4. 

^Clifford Urr, "Will the Real Hypertext Please Stand Up?," 
Computers in Libraries 11 no. 5 (May 1991): 46. 

9 Richard H. Fowler, Susan B. Hancock, and Wendy A. L. 
Fowler, "A Hypermedia System To Explain Library Use. " Computers in 
Libraries 13 no. 11 (December 1993): 14-18. 
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same time as Microsoft Windows 3.0. This product, in effect, was 
the Windows answer to HyperCard. 10 Libraries wasted no time in 
setting up ToolBook applications. The University of the District 
of Columbia Library developad a hypermedia library orientation 
system utilizing itA^ 

Rapid Prototyping 

The concept of rapid prototyping goes back to the mid 1970s 
and is rooted in manufacturing/design and software engineering. 
In the mid 1980s, as i^echnological improvements allowed for more 
powerful rapid prototyping tools, the literature dealing with 
this area increased tremendously. A great deal of this work has 
dealt with software development. Two books that explain the RP 
process in the context of the AskHe project are Rapid Prototyping 
for Object-Oriented Systems by Mark Mullin, published in 1990, 
and Rapid Evolutionary Development: Requirements y Prototyping r 
and Software Creation by Lowell Jay Arthur, published in 1992. 
One example of rapid prototyping of hypertext/hypermedia systems 
are the use of ProtoTymer, a HyperCard RP tool developed at 
Sandia National Labs in collaboration with Stone Design Software 



i^Rami Grunbaum, "Asymetrix Will Unveil Its Long-Secret 
Software This Week," Puqet Sound Business Journal 7 no. 2, (21 
May 1990) : 17. 

Hchet Sarangapani, ''Development and Evaluation of a 
Hypermedia Library Orientation System, " In Proceedings of 
the 12th National Online Meeting Held in New York. Mav 7-9^ 
1991 (Medford, New Jersey: Learned Information, Inc.): 343-348. 
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Inc.^^ ProtoTymer works by fitting itself around a copy of a 
HyperCard application prototype when installed. Then when the 
prototype is run^ ProtoTymer records user activity for later 
analysis. Another is the development of Test Engineering 
Browser, a hypertext knowledge base tool used for spacecraft 
testing at NASA's Jet Propulsion Laboratory • ^3 ^his was done 
through close co-operation by all those involved in the process « 
Today, there are a variety of tools available to rapid prototype 
bypersystems « One that was recommended in 1990 in PC Magazine 
was ToolBook. 

Evaluation Of 
Human Reference Service 



Evaluation of reference service involving human-to-human 
interaction has been an elusive phenomenon. Samuel Rothstein, in 
1964, stated that "The measurement and evaluation of reference 
service has been more often discussed than attempted. " Since 



-^-^Dwight P. Miller and Andrew C. Stone, "ProtoTymer: Human 
Performance Instrumentation for HyperCard Prototyping," In 
Proceedings of the Human Factors Societv 33rd Annual Meeting Held 
in Denver^ Colorado on October 16-20, 1989 (Santa Monica, CA: 
Human Factors Society): 249-253. 

^^irene Hong Woerner, Capturing Flight System Test 
Engineering Expertise: Lessons Learned (Pasadena, CA: California 
Institute of Technology, Jet Propulsion Laboratory, 1991), 273, 
NASA, N91-22790 

^"^Richard Hale Shaw, "ToolBook Puts Windows Development in 
the Hands of Power Users," PC Magazine 9 no. 13 (July 1990): 36. 

•^^Samuel Rothstein, "The Measurement and Evaluation of 
Reference Service," Library Trends 12 (January 1964): 456. 
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then, efforts have been made to better define reference 
transactions as well as to perform evaluations and measurements. 
ANSI Standard Z39.7, revised in 1983, defines a reference 
transaction as "an information contact that involves the use, 
recommendation, interpretation, or instruction in the use of one 
or more information sources, or knowledge of such sources, by a 
member of the reference or information staff, "^^ 

A number of methodologies have been offered in this time 
period. The work of Terrence Crowley and Thomas Childers in the 
late 60s and early 70s concerning unobtrusive measurement of the 
accuracy of reference services, although well cited, has had 
questionable impact on the library-information profession's 
outlook concerning research in this area,^^ Another approach to 
reference service evaluation was put forth by F. Lancaster, 
whose discussion of librarian-customer abilities and attitudes as 
well as environmental factors set up criteria which could be used 
as a basis for measurement • In addition, Terry Weech proposed 
another model involving evaluations by the librarian him/herself. 



i^council of National Library and Information Associations, 
American National Standard for Library and Information Sciences 
and Related Publishing Practices- Library Statistics (New York: 
American National Standards Institute, 1983): 28, ANSI, Z39,7- 
1983 

^^Alvin M. Schrader, "Performance Standards for Accuracy in 
Reference and Information Services: The Impact of Unobtrusive 
Measurement Methodology, " The Reference Librarian no. 11 (Fall/ 
Winter 1984): 197-214. 

l^F. W. Lancaster, "Factors Influencing the Effectiveness 
of Question-Answering Services in Libraries," The Reference 
Librarian no, 11 (Fall/Winter 1984): 95-108. 
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the supervisor, and the customer . In 1990, Lynn Westbrook 

discr.ssed the use of multiple methods in conjunction with each 

other (triangulation) to evaluate reference service. Those 

methods were surveys, observation, interviews, and unob-trusive 
testing, 20 

While there have been efforts involving utilization of some 
of these methodologies, discussion and commentary concerning this 
issue still tends to be the rule rather than the exception. 

The literature concerning the evaluation of reference 
service tends to lack examples of survey instruments. The best 
example found was the Reference Transaction Assessment Instrument 
(RTAI) developed by Marjorie Murfin and Gary Gugelchuk with 
funding and support from Ohio State University in the early 
80s. 21 Murfin and Charles Bunge also collaborated on a project 
to develop an instrument intended to compare the quality of 
academic libraries* reference service. 22 This survey involved 
the use of computer-readable answer sheets, each of which would 
correspond to a reference transaction. Participating libraries 

l^Terry L. Weech, "Who's Giving All Those Wrong Answers? 
Direct Service and Personnel Evaluation." The Reference Librarian 
no. 11 (Fall/Winter): 109-122. 

20Lynn Westbrook, "Evaluating Reference: An Introductory 
overview of Qualitative Methods," RSR (Spring 1990):* 73-78. 

2lMarjorie E. Murfin and Gary M. Gugelchuk, "Development 
and lasting of a Reference Transaction Assessment Instrument," 
College and Research Libraries 48 (July 1987): 314-338. 

22Marjorie E. Murfin and Charles Bunge, "Evaluating 
Reference Service From the Patron Point of View: Some Interim 
National Survey Results," The Reference Librarian no. 11 
(Fall/Winter): 175^182. 
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would submit a collection of answer sheets corresponding to one 
day's transactions. These forms would then be input for data 
analysis • 



Evaluation of Computerized 
Reference Stations 

In the area of evaluation of computerized reference 
systems, a survey was conducted in 1992 at the University of 
Houston by Kathleen Gunning and Kiraberly Speyers-Duran, 23 rphe 
system, called Reference Expert, was evaluated using a paper 
survey which asked users to answer demographic questions as well 
as questions determining familiarity with field of study, ease of 
use, and user preference (Reference Expert or human staff). 

Evaluation of hypermedia reference stations, or for that 
matter, any library hypermedia application, has focused on the 
user satisfaction with the system's interface as opposed to its 
ability to render satisfactory answers to reference questions « A 
major tool for measuring user interface satisfaction is the 
Questionnaire for User Interaction Satisfaction (QUIS) developed 
by Kent Norman and Ben Shneiderman at the University of 
Maryland ,24 m 1988, Joan Cherry and Marshall Clinton conducted 
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^^Kathleen Gunning and Kimberly Speyers-Duran, "Evaluation 
of Reference Expert: An Expert System for Selecting Reference 
Sources," Research in Reference Effectiveness (Chicago: Reference 
and Adult Services Division, American Library Association, 1993): 
Hl-129, 

24 John P, Chin, Virginia A. Diehl and Kent L, Norman, 
"Development of an Instrument Measuring User Satisfaction 
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a survey of user satisfaction with the University of Toronto's 
FELIX online catalog utilizing a variation of the QUIS.25 m 
1990^ Cherry and Clinton conducted a similar survey of five 
online catalogs at universities in Ontario* Both surveys also 
involved the gathering of user demographic data* Patricia 
Thomas, at Kent State University, conducted a study concerning 
satisfaction with the user interface of the Silver Platter ERIC 
CD-ROM on the Macintosh* Her study, conducted in 1993, involved 
use of the QUIS as well as observation and a technique known as 
the "think-aloud protocol* "27 while the QUIS focuses on such 
issues as size and shape (font) of characters, screen layout, 
commands, and task performance, it does not directly deal with 
the system's ability to answer a reference question to a user's 
satisfaction* 

While there are studies of user satisfaction with human 

reference service published in the literature, there is little 

yet that examines satisfaction with computerized reference 

of the Human-Computer Interface," In CHI '88 Held in Washington, 
D*C* on Mav 15-19, 1988 (New York: Association for Computing 
Machinery, 1988): 213-218* 

25joan M. Cherry and Marshall Clinton, "A Profile of User 
Backgrou^ and User Satisfaction With the University of Toronto 
OPAC and the Implications for User Training and User Interfaces," 
In Proceedings of the 52nd Annual Meeting of the American Society 
for Information Science Held in Washington, D*C* on October 30- 
November 2, 1989 (Medford, New Jersey: Learned Information Inc*): 
121-128* 

26joan M* Cherry and Marshall Clinton, "OPACs at Five 
Ontario Universities: A Profile of Users and User Satisfaction," 
Canadian Library Journal * 49 no* 2 (April 1992): 123-133* 

27patricia S* Thomas, "The User Interface of ERIC on the 
Macintosh: A Qualitative Study of Novice Users" (Master's 
research paper, Kent State University, 1993), 23-24* 
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stations. A possible reason for this Is that development of 
these systems has been going on for less than ten years. Much 
the literature dealing with hypertext/hypermedia reference and 
orientation systems concerns development. Implementation, and 
features. That which does evaluate such systems focuses on 
Issues other than the ability to lead one to a satisfactory 
answer to a reference question. 
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PROJECT ASKME 



Before AskMe 



The seeds of AskMe were sewn in the Fall of 1991 when a 
"hypermap" of the Wright Laboratory Tech Librairy was designed 
using the shareware package HYPER. This was for demonstration 
purposes onlyi The hypermap consisted of layouts of sections of 
the library linked with each other as well as linked with 
descriptive text placed in pop-out windows. The source file was 
in ASCII format which was then compiled by HYPER to create the 
hypertext. There were limitations to this environment in that 
complex hypersystems required the creation of very large text 
files that became cumbersome to edit. The size of these files 
exceeded the memory capabilities of screen editors such as PC- 
Edit and TED. One needed to use a word processing program such 
as WordPerfect to handle the file. As a result, debugging was an 
exercise in tossing files between HYPER and WordPerfect. A more 
robust environment was desired. 



E nter ToolBook 



Since the library utilized PCs, HyperCard was out of the 
question. Regardless of that, other projects and activities 
within the library caused the hypermap to be put on hold. 
Interest was rekindled upon discovery, in a journal article, of 
a powerful piece of software called ToolBook. In 1993, serious 
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effort was put forth toward the development of a system that 
would be more than just a "hypermap," The goal now was to design 
a working prototype of a fully functioning hypertnedia-based 
reference station. Further investigation into ToolBook and other 
packages such as HyperWriter and AskSam yielded a decision to go 
with ToolBook as the software platform. 

In July/August of 1993^ the philosophy behind and plans for 
the yet-unnamed reference station were documented. With the 
acqvisition of the ToolBook software in August^ work on the 
reference station began in earnest. 

AskMe 

The name '* AskMe" was chosen out of a desire to invite 
library customers to use the reference station. Inspiration was 
drawn from Valerie Feinman*s scenario in which a sign next to a 
library orientation station says "TRY MEI."28 rp^e name is simple 
and gives the reference station an aura of personality. From the 
beginning, AskMe has been considered a uinque entity able to 
change and grow to meet user needs. At present, the AskMe 
prototype is in reality hypertext and not hypermedia. The reason 
for this is that while library maps were a part of earlier 
versions, they were removed doe to obsolescence resulting from 
movement of staff and resources taking place over a period of 
months. Consequently, the prototype contains text only. However 

28valerie Feinman, "Hypertext and Library Instruction. 
Computers in Libraries . 13 no. 6 (June 1993): 49. 
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the hypermedia nature of AskMe will be restored in the near 
future with the inclusion of updated maps* 

A major feature of AskKe is the "Queiry Bar" which, when 
clicked upon with the mouse, allows a user to type in a word or 
words (ask) and thus be presented with a screen describing a 
library service or facility or a suggestion to utilize tools such 
as the online catalog or the CD-ROMs. As AskMe matures, this 
feature will become more powerful* 

The hypertext nature of AskMe is most evident in the use of 
"buttons" as "nodes" to link screen describing library facilities 
or services* Buttons are placed in both menu formation and 
within other text* Originally "hotwords" or plain text used in a 
node position was going to be used within other text. However, a 
decision was made to use buttons in order to make i^ more evident 
to the user that clicking on the text would lead somewhere else* 
Buttons are colored gray to make them stand out above text and 
background* The only exception to the use of buttons thus far is 
in the index* Here, fields appearing as gray rectangles are used 
so that the text within each field can be arrived at through 
clicking on a button or making a query. Thir is because button 
text cannot be used as a destination in ToolBook* These fields, 
however, function in a manner similar to buttons* In short, it 
has been decided that anything that is gray in the AskMe 
environment can be clicked on in order to go elsewhare* In this 
way, users will not be confused as to what is a node and what is 
not * 
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At the bottom of each AskMe screen, with the exception of 
the Main Menu and Index, are three buttons. One, on the lower 
left, takes the user to the Main Menu screen while another, on 
the lower right, goes to the index. These buttons are labeled 
accordingly. The third button, located in the lower middle, is 
labeled "Go Back." This button navigates backward to the 
previous screen used. Clicking on this button allows us^rs to 
work their way back in order to make other choices at a given 
"level." As mentioned previously, the Main Menu and Index 
screens are the exceptions in that buttons that lead to 
themselves are excluded. 

Most textual information is white on a dark blue background. 
Originally, text was black on a cream-yellow shadowed field on a 
light brown background. However, the cream-yellow was too bright 
for large amounts of text. Light blue-green was experimented 
with and yielded more favorable results. Eventually, for the 
sake of simplicity, the shadowed-f ield appearance was eliminated 
for explanatory text, the text color was changed and the 
background darkened for ease of reading. Presently only title 
text is black on a cream-yellow field. The font or "print style" 
most explanatory text is in a 14-point Couri'^r New bold. Where 
space is an issue, a smaller point size is used but only if 
necessary. Title Text is in a 24-point Arial (non-serif) bold 
font except for the AskMe and Index title text. The AskMe title 
text is the largest of all, in a 60-point Arial bold while the 
title text for the index is :*n a 36 -point Arial bold. Origin- 
ally, all explanatory text was in a 16 to 32-point Arial bold. 
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The Intention was to make the print easy to read for those with 
varying visual capabilities. However based on comments that the 
size of explanatory text was too intimidating and that the text 
did not look like what users may be used to seeing on a computer 
screen, the 14-point Courier New bold font was chosen as a 
satisfactory compromise. 

Besides the Query Bar and the interlinked explanatory text, 
AskHe has two other major features (1) access to in-house dBase 
databases, and (2) connectivity to Internet resources. Pres- 
ently, the Wright Laboratory Technical Library's Journal Holdings 
List is the only dBase database available, but others will follow 
soon. In this capacity, ToolBook acts as a "front-end" for the 
dBase database. When one reaches this front-end screen, he/she 
can click a button to search journal titles, A box is presented 
which allows one to type in the sought-after title. If there is 
a find, that record is presented on the screen, if there is no 
find, a "not found" message appears and the record closest to the 
query is presented. One also has the option to browse titles 
forward or backward by clicking on the appropriate button. This 
is useful if one is not sure of the title or one wishes to look 
for titles beginning with a word or words which describe an area 
of interest (i, e, flight or air power). In the future, subject 
as well as word in title/subject will be made available. 

The other major feature of AskHe is the ability to connect 
to Internet resources originating both within and outside the 
Library, These resources include the Library's CLSI online 
public access catalog, the OhioLink service. The individual 
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catalogs of Wright State University and the University of Dayton^ 
the AFIT (Air Force Institute of Technology) Gateway (which 
includes the AFIT catalog), LC-MARVEL (Library of Congress), and 
the Wright-Patterson AFB Information System. This feature brings 
a tremendous amount of information to the user through access to 
library catalogs, gopher servers, bulletin boards, and other 
resources located around the globe. 
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DATA ANALYSIS 



The Survey Instrument 



The survey instrument is intended to measure the overall 
effectiveness of a given prototype of the AskMe reference 
station. It is meant to be short so that users will not feel 
bogged down in questions and thus not complete the survey* 
Questions 1-4, 6,8 and 9 are answerable by one of several choices 
while Questions 5, 7, 10, and 11 are requests for comments. 
Answering these questions is not a requirement for completion of 
the survey; however it opens the door to further user input that 
is of potentially great value to the developer in the rapid 
prototyping process. 

Questions 1 and 2 are introductory questions. Question 1 
pinpoints first-time users as well as others who have used AskHe 
a few or many times previously. Question 2 asks how first-time 
users learned of AskMe. Besides giving this inf omation, answers 
to Question 2, as well as Question 1, can be used as a basis for 
finding out such things as how first-time users who discovered 
AskMe by seeing it in the library evaluate it as opposed to thos3 
first-time users who learned of it in the base newspaper, the 
Skywriter. 

Question 3 asks if the user found what he/she was looking 
for. Choices are designed to supply a range of alternatives 
including that of neutrality. These alternatives are: finding 
nothing at all, not finding anything relevant to the search but 
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coming upon information useful in other ways, no opinion, finding 
information that is relevant but not exactly what is being looked 
for, and finding exactly what was looked for. This is the 
question whose answers are monitored for a measure of overall 
system effectiveness. 

At this point, the questions automatically branch. In this 
way there is no stumbling over screens to get to the next 
relevant question. Questions 4 and 5 are -r-i^ant to find the 
strengths of the system by asking the user to point out the 
features that led to desired infomation. Questions 6 and 7, on 
the other hand, seek to find the weaknesses of the system by 
asking what kept the user from finding an answer. Questions 4 
and 6 allow the user to identify one or more specific items as 
that which helped or hindered the search. These items are chosen 
because these are either major system features or potential 
trouble spots identified by the developer for close user 
monitoring. In both of these questions, there is an option to 
put "Other" so that strengths or weaknesses not mentioned can be 
identified by users in questions 5 and 7. In Questions 3, 4, 6, 
8, and 9 a "No opinion" option is offered to allow for complete 
neutrality. When a user answers "No opinion" to a question, 
he/she is sent to the next relevant question. 

Questions 8-10 deal with a major issue in hyporsystems . 
That issue is the user's ability to navigate the system. 
Question 8 asks for the degree of ease or difficulty in doing 
this. Question 9 again asks the user to identify one or more 
feature-related reasons for having any difficulty in getting 
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around the system*. These are chosen for the same reason as those 
in Questions 4 and 6. If one answers "Found my way around very 
easily" or "No opinion" to Question 8, he/she is sent to Question 
11. 

Question 11 allows the user to enter any other comments he 
or she may have concerning the system. This opens the door to 
the addressing of particular issues previously not considered 
which can help the developer focus on needed improvements that 
can be made in future versions. Comments that are entered will 
also be available to the developer as soon as a given survey 
response is added to the database. 

The Data 

Text files are created every time a survey is answered. A 
separate ToolBook application acts as a scoreboard containing 
cumulative field-cells corresponding to the multiple-choice 
questions. This template is laid out to look like a survey form 
with field-cells in the position of the answer boxes for each 
choice. As each suirvey is processed, 1 is manually added to the 
amount in each field-cell which corresponds to a chosen box. 
This template, as well as individual survey files can be examined 
at any time. Comments, if any, can be assessed for user 
perception of strengths and weaknesses. For the purposes of this 
study, data is being viewed in terms of the way Question 3 
(dealing with to which degree the user gets or is led to an 
answer) is answered. 
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Results 

There were many challenges to this project once AskMe was 
set up in the Library. Wiring constraints caused the reference 
station to be set up in a comparatively remote part of the 
Reference area. This situation was not changed until mid-May 
when new furniture was installed and new network drops and power 
cables were put in place. Then AskMe could be placed closer to 
the Library entrance. Even with this change, response to the 
survey was very low. There were only nine entries over the two- 
month period. System error was ruled out as a reason due to the 
fact that testing was conducted both before implementation and 
during the data collection period when the lack of responses 
yielded suspicion. At both times the system was functioning 
properly. The low rate may have been due to library customers' 
being too busy with what they had come in to do to answer a 
survey, no mattei how short or concise it may have been. Another 
possible reason is that customers had no desire to fill out "yet 
another survey" as such things are coxmnon in the present on-base 
work environment. Yet another possible reason was made evident 
through one respondent's comment. It stated that he/she "hit the 
control-arrow to go back a word for editing and a new box came 
up. I don't know if what I entered in a previous box is still 
there or not, so I'll try again." In ToolBook, Ctrl-left /right 
arrow will take one back a "page" (screen) or forward one screen, 
depending on which key is pushed. It evidently is also the way 
that one or more editors or word processors allow one to move 
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backward or forward one word to for editing. Since so many 
systems are used on-base, it never occurred to the author that 
this would be a problem, although the possible hitting of Ctrl- 
left/right arrow was considered. Reasons considered for hitting 
these keys were accident or a respondent's knowing ToolBook and 
trying to be "clever". The likelihood of this was thought to be 
low since screen directions were quite explicit and it was 
assumed that anyone who did know ToolBook would be co-operative 
and follow those directions. There may have been instances in 
which users pushed these keys to edit and, upon finding 
themselves in another screen, decided to withdraw from the 
survey. 

Overall, the low number of responses was a tremendous 
disappointment as it was anticipated that this number would be 
higher. In addition, one of the responses had to be thrown out 
due to null responses to several questions. Two others were 
considered questionable since they had null responses to one 
question apiece, one to Questions 1 and the other to Question 4. 
However, after much thought on the matter, it was decided to keep 
these responses because (1) Question 3, the central question, was 
answered by both, (2) One gave a negative response to Question 3 
and answered Question 6, and (3) One of these responses contained 
the comment concerning the Ctrl-arrow editing situation. The 
reason for null responses to these questions is probably that a 
user tried to edit a comment response using Ctrl-arrow and backed 
into a multiple-choice screen. The survey is set up to so that 
when one enters a page (screen) any answer that was there is 



27 



ERIC 



cleared so that no one can see anyone else's answer (s); thus if 
someone keeps hitting Ctrl-arrow going backward and forward 
he/she will eliminate his/her own answers as well. If the user 
decides to complete the survey^ those responses, including null 
ones, will be put to disk. 

Nevertheless, here is the final tally from the scoreboard 
shown in the tables below: 



Table 1. — Question 1. How many times have you used this 
system previously? 



Never 1-2 times 3-5 times More then 5 times 



Users 



Table 2. — Question 2. If a first time user, how did you first see 
or hear about the system? 

Word of Nf wsletter/ Demonstration Discovery 
mouth Skywriter in in 

article Library Library 



Users 



Table 3 . — Question 3 . Did you find what you were looking for by 
using this system? 

No, not No, but No opinion Yes, but not Yes, exactly 

at all found useful exactly what what I was 

unrelated i was looking looking for 

information for 



Users 
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Table 4. — Question 4. If you answered "Yes", which system feature (s) led you to 

what you were looking for? (You may click on more than one.) 



Query bar Journal Holdings Descriptions of Connections Other No opinion 
list search screen Library resources to other 
and services systems 



Users 1 1 2 10 0 



Table 5. — Question 6. If you answered "No", which system weakness (es) kept you 
from finding what you were looking for? (You may click 
on more than one.) 





Repeated use 


Limitations 


Not enough 


Too much 




of query bar 


in Journal 


detail in 


detail in 




yielded nothing 


Holdings List 


descriptions 


descriptions — 






searching 


of Library 


cluttered or 








resources 


confusing 








and services 




Users 


0 


0 


0 


0 


Table 


5. (continued) 










Connections not 


Too many 


Other No 


opinion 




available to 


connections 








more systcims 


available, 










hard to choose 






Users 


0 


0 


0 


1 



Table 6. — Question 8. How easy or difficult was it to "find your way around" in 
the system? 



Could not find Had great No opinion Found my way Found my way 

my way, felt difficulty around with around very 

quite lost finding my way a little work easily 



Users 0 0 15 2 
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Table 7. 


— Question 9 


If you any difficulty finding your way around, what, in 




Too many 


Buttons did Text was Other No opinion 




screens or 


not lead confusing 




levels 


where I 






wanted to go 


Users 


0 


2 3 10 



Th.>re were three known first-time users who responded to the 
survey. Of these three, two answered that they discovered AskMe 
in the Library while one answered that he/she discovered it 
through word of mouth. This indicates that at least some 
customers were curious enough to investigate the new system. The 
pivotal question. No, 3 was responded to negatively by one 
individual, and positively by six while one expressed no opinion. 
The negative respondent said that he/she found nothing. Of the 
positive respondents, two said that they did not find exactly 
what they were looking for, however three said that they did. 
The Query Bar helped one person while descriptions of resources 
and services helped the other two and the Journal Holdings List 
and connections to other systems helped one each. "No opinion" 
was given in the case of the negative response. Since one user 
answered "No opinion" on Question 3, he/she was branched to 
Question 8, Question 8 had five persons answer that they found 
their way around with a little work while two found their way 
around very easily. One answered "No opinion," The reason for 
any stumbling was confusing text in three cases. In addition, 
one respondent also cited buttons not leading to where he/she 
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wanted to go. One person cited the buttons as a sole reason 
while one ansvrered "other" but did not comment. 

Comments were given by two respondents to the suirvey. One, 
answering Question 10, stated that there were "no instructions on 
backing out of [UnCover] . " Another, answering Questions 5 and 
10, said, "looked for specific journals, none available here." 
(Question 5) and "I tried to find specific journal titles. Would 
be useful if I could get just a list of titles, rather than a 
dump of details on each title. I wanted to see if I had the 
title wrong, but close. I just wanted titles, not all the other 
info at this point." (Question 10). 
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CONCLUSION 

In this particular setting and within the allotted time 
frame, the survey results alone have not proven to be a good 
evaluation of AskMe's effectiveness due to the low number of 
responses. However, that does not lead to a conclusion that the 
method is useless by any means. Perhaps in a setting such as an 
large academic library where there are more available resources 
such an approach, used alone or combined with others, may yield 
better results. At this stage of the rapid prototyping process 
for AskMe, given that there is insufficient staff or space to 
perform other types of formal studies based upon in-house or 
laboratory observation, the best approach to evaluate AskMe's 
effectiveness lies in personal communication between developer, 
other Library staff, and user. All must have a sense of 
ownership within the process. It is the conclusion of the 
author, based upon the information gathered and the experience 
gained over the last year, that the rapid prototyping approach to 
the building of the hypermedia workstation AskMe, within the 
context of the Wright Laboratory Technical Library, lies outside 
the established paradigm of conventional research. It involves 
constant change based upon constant communication. This can take 
the form of casual observation and the seeking out of comments 
and suggestions from users and staff both verbally and 
electronically . 



ERIC 



3S 



FUTURE PLANS 



32 



The immediate future, as previously mentioned, calls for the 
placement of library floor maps onto AskNe. Decreased staff, 
increase in workload, and the devotion of time and energy to this 
project has delayed the updating of these maps until the present 
time. Plans also call for linkage to a compiled dBase program 
allowing the users to search journal holdings by title, subject, 
or both. Also, users will be able to search journal titles sub- 
scribed to (through the Library) by offices in remote locations. 
In addition, bibliographies concerning issues sufjh ISO 9000 and 
other "hot topics" may be placed on board. Also, plans call for 
an upgrade to Multimedia ToolBook which, combined with hardware 
upgrades, will allow users to view video and other multimedia 
productions. The ultimate goal is to make AskMe available to 
users outside the physical boundaries of the library. 
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Electronic Survey Instrument 



1. How many times have you used this system previously? 

^Hever 

1-2 times 

3-5 times 

^more then 5 times 

2. If a first time user, how did you first see or hear about the 
system? 

Word of mouth 

Newsletter/Skywriter article 

Demonstration in Library 

^Discovery in Library 

3. Did you find what you were looking for by using this system? 

^No, not at all 

^Ho,, but found useful unrelated information 

^No opinion 

^Yes, but not exactly wuat I was looking for 

Yes, exactly what I was looking for 

4. If you answered "Yes", which system feature (s) led you to what 
you were looking for? (You may click on more than one.) 

Query bar 

Journal Holdings List search screen 

^Descriptions of library resources and services 

^Connections to other systems 

Other 

^No opinion 

5. Please feel free to explain your choice in box. 



6. If you answered "No", which systenr weakness (es) kept you from 
finding what you were looking for? (You may click on more 
than one, ) 

^Repeated use of query bar yielded nothing. 

^Limitations in Journal Holdings List searching 

^Not enough detail in descriptions of library 

resources and services 
^Too much detail in descriptions — cluttered or 

confusing 

^Connections not available to more systems 

Too many connections available, hard to choose 

^Other 

^No opinion 

7. Please feel free to explain your choice in box. 



8. How easy or difficult was it to "find your way around" in the 
system? 

^Could not find my way, felt quite lost. 

Had great difficulty finding my way 

No opinion 

^Found my way around with a little work 

^Found my way around very easily 



9. If you any difficulty finding your way around, what, in your 
opinion, caused this? 

Too many screens or levels 

^Buttons did not lead to where I wanted to go 

Text was confusing 

Other 

^No opinion. 

10. Please feel free to explain your choice in box. 



11. Additional Comments (if any): 



Thank you very much for your time and co-operation. 
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